Using latent semantic analysis to measure coherence in essays by foreign language learners?

This paper compares human and LSA-based coherence ratings of 223 essays written by learners of English as a foreign language at various levels of linguistic proficiency. While the LSA-based ratings correlate positively – as expected – with a measure of lexical variety, a negative correlation was found between these ratings and the human grading. This is interpreted as resulting from specific writing strategies used by authors of different proficiency levels but also from the fact that human graders were sensitive to a wider range of linguistic cohesive devices.